Disputation
16 April 2024
University of Mannheim
Which methods can we use to classify data from open-ended survey questions?
Can we leverage these methods to make empirical contributions to substantial questions?
Motivation:
1️⃣ Increase in methods to collect natural language (e.g., smartphone surveys with voice technologies) requires the evaluation of automated, ML-based methods.
2️⃣ Special structure of open-ended survey answers (e.g., shortness, lack of context) requires the testing of traditional and recent methods, e.g., word embeddings.
Table 1. Overview of methods for classifying open-ended survey responses. Source: Own depicition.
| Study 1 | Study 2 | Study 3 |
|---|---|---|
| How valid are trust survey measures? New insights from open-ended probing data and supervised machine learning | Open-ended survey questions: A comparison of information content in text and audio response formats | Asking Why: Is there an Affective Component of Political Trust Ratings in Surveys? |
Landesvatter, C., & Bauer, P. C. (2024). How Valid Are Trust Survey Measures? New Insights From Open-Ended Probing Data and Supervised Machine Learning. Sociological Methods & Research, 0(0). https://doi.org/10.1177/00491241241234871
Background: ongoing debates about which type of trust survey researchers are measuring with traditional survey items (i.e., equivalence debate cf. Bauer & Freitag 20181)
Research Question: How valid are traditional trust survey measures?
Operationalization via two classifications: share of known vs. unknown others in associations (I), sentiment (pos-neu-neg) of associations (II)
Supervised classification approach:
Data: U.S. non-probability sample; n=1,500
Landesvatter, C., & Bauer, P. C. (February 2024). Open-ended survey questions: A comparison of information content in text and audio response formats. Working Paper submitted to Public Opinion Quarterly.
Background: requests for spoken answers are assumed to trigger an open narration with more intuitive and spontaneous answers (e.g., Gavras et al. 20221)
Research Question: Are there differences in information content between responses given in voice and text formats?
Experimental Design: random assignment into either the text or voice condition
Landesvatter, C., & Bauer, P. C. (March 2024). Asking Why: Is there an Affective Component of Political Trust Ratings in Surveys?. Working Paper submitted to American Political Science Review.
Background: conventional notion stating that trust originates from informed, rational, and consequential judgments is challenged by the idea of an “affect-based” form of (political) trust (e.g., Theiss-Morse and Barton 20171)
Research Question: Are individual trust judgments in surveys driven by affective rationales?
Large Language models (LLMs) facilitate the accessibility and implementation of semi-automated methods.
traditional semi-automated methods, such as supervised ML, are helpful and appealing, but they require sufficient and high-quality training data (i.e., labeled examples)
E.g., Study 1: Random Forest with 1,500 labeled examples versus BERT
this can be a challenge for survey researchers when surveys don’t provide thousands of documents
LLMs allow researchers to access and leverage great capabilities without having to build complex systems from scratch
Fine-tuning pre-trained models can be valuable for classifying domain-specific data.
Increasing number of possibilities to reduce manual input to a minimum.
The final decision for one of the approaches depends on:
difficulty of the given task (e.g., general versus specific codes)
size of the available dataset (e.g., n, splits by experimental conditions)
structure of the open answers (e.g., length, amount of context → this depends on the question design)
the amount and state of previous research (e.g., available code schemes)
desired accuracy and desired transparency
available resources (e.g., human power, computational power (GPU), time resources)
Landesvatter: Methods for the Classification of Data from Open-Ended Questions in Surveys